Method for generating an augmented representation of a real environment, corresponding device, computer program product, and computer-readable carrier medium

ABSTRACT

The disclosure relates to a method for generating an augmented representation of a real environment, called an augmented reality representation. Such a method includes obtaining, as a function of said real environment, a virtual boundary dividing the real environment into two spaces, respectively called an activity space and a mixed space. The augmented reality representation is generated by obtaining a first part of the augmented reality representation, called an activity part, corresponding to a representation of at least one part of the activity space. Thereafter, obtaining a second part of the augmented reality representation, called a mixed part, corresponding to an augmented representation of at least one part of the mixed space, in which at least one virtual content is combined with the representation of at least one part of the mixed space.

REFERENCE TO RELATED EUROPEAN APPLICATION

This application claims priority from European Patent Application No.18305267.9, entitled “METHOD FOR GENERATING AN AUGMENTED REPRESENTATIONOF A REAL ENVIRONMENT, CORRESPONDING DEVICE, COMPUTER PROGRAM PRODUCT,AND COMPUTER-READABLE CARRIER MEDIUM”, filed on Mar. 13, 2018, thecontents of which are hereby incorporated by reference in its entirety.

FIELD OF THE DISCLOSURE

The present disclosure relates to the field of augmented realityapplications. More particularly, the present disclosure relates to atechnique for generating an augmented representation of a realenvironment.

BACKGROUND

The present section is intended to introduce the reader to variousaspects of art, which may be related to various aspects of the presentdisclosure that are described and/or claimed below. This discussion isbelieved to be helpful in providing the reader with backgroundinformation to facilitate a better understanding of the various aspectsof the present disclosure. Accordingly, it should be understood thatthese statements are to be read in this light, and not as admissions ofprior art.

In augmented reality applications taking place in an uncontrolledcomplex real-world environment, for inserting virtual objects in mixedreality in a user's living room, it is recommended to understand andanalyse the layout of the real scene, but also its radiometry, thegeometry and appearance of the virtual objects, the user position, andthe purpose of the application itself.

In an augmented reality representation provided to the user, it mayhappen that a virtual content obscures a real-world object close to theuser in the real world environment. To overcome this problem, theapproach generally adopted by prior art solutions consists in modifyingin real-time the display of a virtual content, when it is detected thatthis virtual content is likely to obscure the view of the user. In thatway, the user can see nearby real-world objects and avoid collision whenhe moves around. For example, in document US 2016/0027212 A1, datacaptured by sensors embedded on a head-mounted display worn by a userare used to compute in real-time distances between the user and nearbyreal-world objects. If a distance between the user and a real-worldobject becomes lower than a predetermined threshold when the user movesaround in the real-world environment, any virtual content overlaid onthis real-world object is dimmed or faded out, thus allowing the user tobecome aware of the presence of this object. As an alternative, aproblematic virtual content may also be reduced in size, moved elsewherein the augmented reality representation, or even removed from theaugmented reality representation, as described for example in documentWO 2013/052855 A2.

Although effective for ensuring the safety of the user, these solutionsof prior art make the mixed scenario less realistic, and compromise thequality of the user experience. Indeed, the proposed solutions relyingon modifying in real-time a problematic virtual content (by dimming it,fading it out or reducing its size for example) cannot satisfy the goalof consistent perception of both virtual content and real scene. Inaddition, such a real-time adjusting of virtual content also requiresimportant computing power, which may monopolize a significant part ofthe resources of the processor in charge of generating the augmentedreality representation.

It would hence be desirable to provide a technique for generating anaugmented representation of a real environment that would avoid at leastone drawback of the prior art.

SUMMARY

The present disclosure pertains to a method for generating an augmentedrepresentation of a real environment, called an augmented realityrepresentation. The real environment comprises at least one real object.Such a method comprises:

-   -   obtaining, as a function of data associated with objects of the        real environment, a virtual boundary dividing said real        environment into two spaces, respectively called an activity        space and a mixed space;    -   generating said augmented reality representation, comprising:        -   obtaining a first part of said augmented reality            representation, called an activity part, corresponding to a            representation of at least one part of said activity space;            The activity part is free of virtual content;        -   obtaining a second part of said augmented reality            representation, called a mixed part, corresponding to an            augmented representation of at least one part of said mixed            space, in which at least one virtual content is combined            with said representation of at least one part of said mixed            space.

According to the proposed technique, the augmented realityrepresentation is thus divided in two parts: a first part correspondingto a non-augmented representation of the real environment, and a secondpart corresponding to an augmented representation of the realenvironment. In that way, a user (who wears a head-mounted device forexample) can moves safely within the activity space in the realenvironment, since the corresponding activity part of the augmentedreality representation is free of virtual content (therefore, there isno risk that a user moving within the activity space bumps into or tripover real-world object because of a virtual object obscuring his view inthe augmented reality representation). In addition, the quality of theuser experience is better than with the existing techniques since theuser can move freely in the activity space with no additional need todim or fade out the display of virtual content to make the user seenearby real-world object: the mixed scenario thus stays realistic, witha permanent and consistent perception of both virtual and real content.The computing power required in previous art solutions to computereal-time distances between the user and real-world objects, and to dimor fade out virtual objects overlaid on them when the user gets tooclose, is also saved with the proposed technique.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present disclosure can be better understood withreference to the following description and drawings, given by way ofexample and not limiting the scope of protection, and in which:

FIG. 1 is a flow chart for illustrating the general principle of theproposed technique for generating an augmented representation of a realenvironment, according to an embodiment of the present disclosure;

FIG. 2 shows an example of the division of a real environment into anactivity space and a mixed space according to an embodiment of thepresent disclosure;

FIG. 3 shows an example of an augmented reality representation of thereal environment introduced in FIG. 2, according to an embodiment;

FIG. 4 is a schematic block diagram illustrating how the proposedtechnique may be implemented, according to an embodiment of the presentdisclosure;

FIG. 5 illustrates how an initial position of the user may be taken intoaccount to obtain a virtual boundary, according to an embodiment of thepresent disclosure;

FIG. 6 depicts an example of a virtual boundary adapted to encompass agiven real object, according to an embodiment of the present disclosure;

FIG. 7 shows how a virtual ground area size of a default virtual scenemay be obtained, according to an embodiment of the present disclosure;and

FIG. 8 is a schematic block diagram illustrating an example of anapparatus for generating an augmented representation of a realenvironment, according to an embodiment of the present disclosure.

The components in the figures are not necessarily to scale, emphasisinstead being placed upon illustrating the principles of the disclosure.

DETAILED DESCRIPTION General Principle

The general principle of the present disclosure relies on a specifictechnique for generating an augmented representation of a realenvironment, also referred to as augmented reality representationthroughout the document. Such an augmented reality representation isprovided to a user by means of a rendering device such as for example ahead-mounted display, a mobile phone (smartphone), or a tablet.

As it will be described more fully hereafter with reference to theaccompanying figures, it is proposed in one aspect of the presentdisclosure to virtually divide the real environment into two distinctspaces: one space which may be assimilated as a safety zone within whicha user can move around safely while viewing an augmented representationof the real environment, and another space specifically defined as theusable space for the insertion of the virtual content in the generatedaugmented reality representation.

This disclosure may, however, be embodied in many alternate forms andshould not be construed as limited to the embodiments set forth herein.Accordingly, while the disclosure is susceptible to variousmodifications and alternative forms, specific embodiments thereof areshown by way of example in the drawings and will herein be described indetail. It should be understood, however, that there is no intent tolimit the disclosure to the particular forms disclosed, but on thecontrary, the disclosure is to cover all modifications, equivalents, andalternatives falling within the scope of the disclosure as defined bythe claims. For example, the term “obtain” used with respect to a givencharacteristic throughout the disclosure (such as obtaining a virtualboundary, an activity part, a mixed part, etc.) is to be understood asencompassing both the situations where this characteristic is determinedby the device implementing the proposed technique itself, or receivedfrom an external device. In the drawings, alike or similar elements aredesignated with identical reference signs throughout the several viewsthereof.

While not explicitly described, the present embodiments and variants maybe employed in any combination or sub-combination.

FIG. 1 is a flow chart for explaining the method for generating anaugmented representation of a real environment, according to the generalprinciple of the present disclosure.

At step 11, a virtual boundary dividing the real environment into twospaces, respectively called an activity space and a mixed space, isobtained.

At step 12, an augmented representation of the real environment (calledan augmented reality representation) is generated, as a function of thevirtual division previously obtained at step 11. More particularly,generating the augmented reality representation comprises:

obtaining a first part of said augmented reality representation, calledan activity part, corresponding to a representation of at least one partof said activity space;

obtaining a second part of said augmented reality representation, calleda mixed part, corresponding to an augmented representation of at leastone part of said mixed space, in which at least one virtual content iscombined with said representation of at least one part of said mixedspace.

FIGS. 2 and 3 illustrate, with an example, the effect of carrying outthe proposed technique for generating an augmented representation AR ofa real environment RE.

In this example, the real environment RE comprises four objects (R1, R2,R3 and R4) lying on the ground, in a corner of a room formed by twowalls W1 and W2, as illustrated on FIG. 2 in a perspective view.Although represented by simple shapes on FIG. 2, R1, R2, R3 and R4 arereal-world objects (such as, for example chairs, table, etc.). In thecontext of the disclosure, the term “object” must be understood in abroad sense: an object may refer to a static or mobile object, and aperson (other than the user viewing the augmented representation) is forexample considered as an object. A user U, for whom the augmentedreality representation is intended, stands within this real environmentRE, for example at a position P. The user U is equipped with a deviceallowing him to view the augmented reality representation. For example,he wears a head-mounted display or special glasses on which theaugmented reality representation is rendered, or he holds a mobile phoneor a tablet on the screen of which the augmented reality representationis displayed.

As explained in relation with step 11 of FIG. 1, a virtual boundary VBdivides the real environment into two separated spaces respectivelycalled an activity space AS and a mixed space MS. Although representedby a simple two-dimensional dotted line projected on the ground in FIG.1, the virtual boundary VB may actually take the form of athree-dimensional surface dividing the three-dimensional space of thereal environment, such as for example the surface of a half sphere, orof a cylinder (virtually extending from floor to ceiling for example, incase of an indoor environment). The way that the virtual boundary VB isobtained will be described later in the document.

FIG. 3 shows an example of an augmented reality representation ARgenerated according to the proposed technique, and provided to the user(rendered for example in real-time on a head mounted display worn by theuser). In this example, the augmented reality representation ARcorresponds to a representation of the real environment RE as it may beseen by a user standing at the position P shown in FIG. 2 and facing thewall W2. As it may be noticed on FIG. 3, this representation is an“augmented representation” since three virtual objects V1, V2 and V3(which are not real-world objects, and which thus do not exist in thereal environment RE) have been added to the generated AR representation.In other words, a virtual scene comprising the virtual objects V1, V2and V3 has been mixed with the real scene, to form the augmented realityrepresentation AR. According to the proposed technique, and as describedin relation with step 12 of FIG. 1, the virtual scene is inserted in amixed part MP of the augmented reality representation AR, correspondingto an augmented representation of at least one part of the mixed spaceMS of the real environment RE. Conversely, no virtual object is insertedin an activity part AP of the augmented reality representation AR,corresponding to a representation of at least one part of the activityspace AS of the real environment RE (the activity part AP thuscorresponds to a non-augmented representation part).

As a result, the activity space AS can be considered as a safety zonewithin which the user can move around freely and safely while viewingthe augmented representation of the real environment, without risk ofbumping into or tripping over real-world object because of a virtualobject obscuring his view. For example, referring back to FIG. 2, aslong as user U is located in the activity space AS, real-world object R2within the activity space AS will not be hidden to the user's sightbecause of a virtual object obscuring his view in the augmented realityrepresentation, since the proposed technique prevents such a virtualobject to be displayed within the corresponding activity part AP of theaugmented reality representation.

When compared with existing solutions described in prior art, theproposed technique provides an enhanced user experience. Indeed, theuser can move freely in the activity space, with no additional need tomodify in real-time the display of some virtual contents (for example bydimming them or fading them out) to make the user see nearby real-worldobject. The mixed scenario (the aim of which is usually to blendseamlessly together virtual and real objects) thus stays realistic, witha permanent and consistent perception of both virtual and real content.In addition, the computing power required in prior art solutions tocompute in real-time such modifications of the display of virtualcontents is also saved, and may be reallocated to further improve thequality of the augmented reality experience. Finally, because the spacededicated to the user (i.e. the activity space) and the space usable forthe insertion of the virtual scene (i.e. the mixed space) are clearlydistinct spaces, the generated augmented representation is less exposedto undesirable graphic artefacts, due to poorly modelled interactionsbetween virtual content and human, that sometimes appear in augmentedrepresentation generated by prior art solutions when a user breaks intothe field of virtual events. The user experience is thus even moreenhanced with the proposed technique.

DESCRIPTION OF VARIOUS EMBODIMENTS

General Overview

FIG. 4 is a schematic functional block diagram illustrating how thevirtual boundary dividing the real environment into an activity spaceand a mixed space may be obtained, and how the augmented realityrepresentation may then be generated, according to different embodimentsof the disclosure.

Various data are first acquired (401) by various sensors, such aslocalizing sensors (e.g. Global Positioning Systems such as GPS orequivalent, Inertial Measurement Unit including gyroscope andaccelerometers), cameras (e.g. color and/or depth cameras), microphones,etc. These sensors are preferentially embedded directly on a device forgenerating and rendering the augmented reality representation, but othersensors arranged in the real environment may also optionally be used (inthis case, the data acquired by these other sensors are obtained by thedevice for generating the augmented representation through communicationmeans).

The acquired data are classified into two main categories: dataassociated with the user (402), and data associated with the real scene(403) (the real scene being the part of the real environment currentlyanalysed).

Data associated with the user (402) include but are not limited to theposition and orientation of the user within the real environment. Thesedata may be processed (alone or by cross-analysis, in combination withdata associated with the real scene) to deliver user characteristics(404), such as user's height or user's current posture (lying, seated,standing posture).

Data associated with the real scene (403) include but are not limited todata acquired by color and depth cameras, which allow performingthree-dimensional reconstruction of the real scene in real-time(alternatively, a previously determined 3D reconstruction of the realscene may be imported directly). These data associated with the realscene are processed to perform scene classification (405) and sceneanalysis (406).

The scene classification (405) allows determining a category of realscene. For example, the real scene may be classified as an indoor or asan outdoor scene. The real scene may also be classified in a moredetailed manner, for example as a living room scene, a bedroom scene,and so on, in case of an indoor scene.

The scene analysis (406) is carried out by means of various existingimage processing techniques, including image segmentation techniques andrecognition techniques, and allows the detection of a layoutrepresentation of the real scene (407). For example, considering anindoor scene, the layout representation may comprise the ground floor,the bounding walls, the ceiling, and the segmented static objects.Considering an outdoor scene, the layout representation may be limitedto the ground floor and segmented static objects.

The data associated with the real scene (403), which are acquiredperiodically, also allow the detection of moving objects within the realscene (408).

All the functional blocks previously described (401 to 408) pertain tocommon image processing techniques that may be implemented in prior artsolutions for generating augmented reality representation of a realenvironment.

However, according to the present disclosure, the information obtainedas an output of these functional blocks are used in relation with threenew functional blocks for implementing the general principle of theproposed technique. Two of these blocks (obtaining initial virtualboundary (409) and boundary adaptation (410)) pertain to obtaining avirtual boundary (as described in relation with step 11 of FIG. 1),while the third one (managing virtual content (411)) pertains to thegeneration of the augmented reality representation as a function of thevirtual boundary previously obtained (as described in relation with step12 of FIG. 1).

The way these functions are implemented in different embodiments of theproposed technique is detailed hereinafter.

Obtaining Initial Virtual Boundary, and Boundary Adaptation

Obtaining the virtual boundary dividing the real environment into anactivity space and a mixed space is an aspect of the present disclosure.

According to an embodiment, the real environment comprises at least onereal object and obtaining said virtual boundary comprises obtaining atleast part of said virtual boundary as a function of said at least onereal object.

In particular, according to an embodiment, said virtual boundary isobtained as a function of a shape of said at least one real object.

In that way, the virtual boundary can be adapted so that a real objectbelongs totally to one of said activity space or mixed space.

In one embodiment, said at least one real object is a moving object, andsaid at least part of said virtual boundary is obtained dynamically, asa function of said at least one moving real object.

In that way, the virtual boundary can be adapted to take account of amoving object.

In one embodiment, obtaining said virtual boundary comprises obtainingat least part of said virtual boundary as a function of a category ofsaid real environment.

In that way, the virtual boundary can more particularly be adapteddepending on whether said real environment corresponds to an indoorenvironment or an outdoor environment.

According to an embodiment, obtaining said virtual boundary comprisesobtaining at least part of said virtual boundary as a function of ascale difference between said real environment and a default virtualscene comprising said at least one virtual content.

In that way, a piece of data representative of a size of a defaultvirtual scene may be compared to a dimension with respect to the realenvironment to adapt the virtual boundary.

In another embodiment, obtaining said virtual boundary comprisesobtaining at least part of said virtual boundary as a function of aninitial position of a user of the augmented reality representationwithin said real environment.

Thus, the initial position of the user within the real environment istaken into account, in addition to the real environment itself, toobtain the virtual boundary. In that way, an initial neighbourhood ofthis initial position may be included in the activity space.

In that way, the user may also be invited to move if the systemconsiders that the current location will not provide an augmentedreality experience of sufficient quality. Such a situation may happen,for example, if the user's initial position is too close to a boundingwall of the real environment, which would constraint too much theactivity space available to the user, and lead to a degraded user'sexperience.

According to an embodiment, obtaining said virtual boundary comprisesobtaining at least part of said virtual boundary as a function of acurrent height and/or a current posture of the user of the augmentedreality representation.

In that way, the virtual boundary may be obtained, for example,depending on whether the user is standing, seated or lying. Indeed, aseated user may not need as much as activity space as a standing user.More mixed space may then be dedicated to augmented representation for aseated user than for a standing user, for example.

The previously introduced embodiments regarding obtaining the virtualboundary are now further detailed.

The determination of the virtual boundary dividing the real environmentinto an activity space and a mixed space is performed as a function ofthe real environment, and may be carried out in two phases: a firstphase for obtaining an initial virtual boundary, and a second phase forrefining or adapting the previously obtained initial virtual boundary.

During the first phase, according to one embodiment, the initial virtualboundary is obtained by using a distance parameter as a space divisionparameter. Such an embodiment is described in relation with FIGS. 5 and6, showing a top view of the real environment RE of FIG. 2, in whichvirtual objects V1, V2 and V3 have been virtually added. As illustratedin FIG. 5, a distance parameter dr is used as a radius to generate theinitial virtual boundary surface VB′ from a reference position P1 withinthe real environment. Again, for purpose of simplification, only theprojection of the virtual boundary on the ground is shown on the figure(circle in dotted line), but the virtual boundary preferably takes theform of a three-dimensional surface, such as for example the surface ofa half sphere, or of a cylinder. According to a particular feature, thedistance parameter dr (and thus the virtual boundary VB′) is for exampleobtained as a function of a category of the real scene, as obtained asan output of a scene classification module such as block 405 previouslydescribed in relation with FIG. 4. More particularly, the distanceparameter dr is determined depending on whether the real scene is anoutdoor or an indoor scene. Indeed, in an outdoor scene, there isusually more available space than in an indoor scene, and a largeractivity space may be allocated for a user to move around and experiencethe augmented reality representation. For example, the distanceparameter dr may be set to one meter in an indoor real environment, andto three meters in an outdoor real environment. User may also furthertune the distance parameter dr manually.

Determining the distance parameter dr, and thus the initial virtualboundary VB′ may also take account of various users' characteristics.Thus, according to an alternative or complementary feature, the heightand/or the posture of the user may be considered when obtaining theinitial virtual boundary. For example, a seated or lying user is lesslikely to move around than a standing user. Thus, the distance parameterdr may be set to a lower value for a seated or lying user than for astanding user. In a similar way, the distance parameter dr may be set asa function of the height of the user (for example to one time the heightof the user in an indoor environment, and to two or three times theheight of the user in an outdoor environment).

According to an embodiment, the reference position P1 from which theinitial virtual boundary is obtained may be computed automatically as afunction of the result of the analysis of the real scene, thanks to thedata acquired by the various sensors placed in the augmented realitydevice and potentially in the real environment itself. In particular,this reference position may be determined as a position considered asappropriate for viewing a mixed scenario in good conditions, i.e. aposition at least at distance dr from any significant real obstacle(such as walls W1 and W2) which could limit the effective size of theactivity space.

Alternatively, in another embodiment, an initial position of the userwithin the real environment may be used as a reference position toobtain the initial virtual boundary. Once obtained, the virtual boundaryis no more dependent on the position of the user (and the virtualboundary is kept when the user moves around). When the virtual boundaryis obtained as a function of an initial position of the user, the usermay be invited to move to a better position for obtaining the virtualboundary. For example, as illustrated in relation with FIG. 5, a userlocated at position P0 may be considered as too close from the boundingwall W1, since distance dw to the wall W1 is below the distanceparameter dr. At position P0, the wall W1 would indeed limit theactivity space available to the user, and the user experience could thusnot be optimal. In such a case, according to a particular feature, somevirtual cues (e.g. directional arrows) are displayed on the renderingdevice used to view the augmented representation, to advise the user tomove further away from the wall W1 until the distance is large enough(i.e. above the distance parameter dr) for an optimal user experience.

During a second phase, according to an embodiment, the previouslyobtained initial virtual boundary is refined or adapted to take accountof other characteristics of the real environment. More particularly, atleast part of the virtual boundary may be obtained as a function of areal object present within the real environment. In that way, if a realobject is located on the initial virtual boundary, the virtual boundarycan be adapted so that said real object belongs totally to one of theactivity space or mixed space. For example, in the situation illustratedin relation with FIG. 2, real-world object R1 is located on the virtualboundary VB, which could cause some problem when generating theaugmented reality representation (in particular because some virtualobject could be blended with the part of R1 remaining in the mixedspace). As illustrated in relation with FIG. 6, at least a part VB1 ofthe virtual boundary VB is thus adapted as a function of the shape ofreal object R1, so that this object belongs totally to the activityspace AS. An oriented bounding box of the segmented object R1, asobtained as an output of the layout detection module, can notably beemployed to carry out the boundary adaptation (for example by using theprojected edges of the bounding box on the ground). Other situations mayrequire virtual boundary adaptation. More particularly, according to anembodiment, at least part of the virtual boundary is obtaineddynamically, as a function of a moving real object. Such an embodimentis particularly interesting when considering outdoor environment, wheredynamic objects such as pedestrians, cars, bicycles are likely to emergein the scene at any time. Thus, according to a particular feature, thevirtual boundary is adapted automatically to encompass dynamic (i.e.moving) objects that move across the boundary. In that case, the outputof detection of dynamic objects is used in addition to the output of thelayout detection module to carry out the boundary adaptation.

Augmented Reality Representation Generation

Once the virtual boundary dividing the real environment into an activityspace and a mixed space has been obtained, the augmented realityrepresentation can be generated. According to the present disclosure,generating the augmented representation implies managing the virtualcontent, so that the generated augmented reality representation complieswith certain particular constraints, the main of which being that allthe virtual content has to be added only within a given part of therepresentation; i.e. the mixed part associated with the mixed space ofthe real environment. This differs from the approach of conventionalsolutions, wherein the whole space in the field of view of the user ispotentially usable for inserting some virtual objects.

According to an embodiment obtaining said mixed part of the augmentedreality representation comprises scaling the at least one virtualcontent as a function of at least one size difference between a defaultvirtual scene comprising said at least one virtual content and said realenvironment in said mixed space.

In that way, by comparing a default virtual scene with somecharacteristics of the real environment, the scale of the virtualcontent may be adapted to provide the best user's experience accordingto said given real environment.

According to a particular feature, said at least one virtual content isscaled as a function of a difference between a virtual ground area sizeof said default virtual scene and a real ground area size of said realenvironment in said mixed space.

According to an embodiment, at least one virtual content in said mixedpart is determined by taking account of a representation of at least onereal content in said activity part.

In that way, although activity and mixed part are dissociated parts, avirtual object in the mixed part may be rendered not only according toits occlusion with real object belonging to mixed part, but alsoaccording to its occlusion with real object belonging to the activitypart.

According to another embodiment, the method for generating an augmentedrepresentation of a real environment further comprises alerting a userof the augmented reality representation when a distance between saiduser and the virtual boundary is lower than a predetermined threshold.

In that way, the user may be warned if he is about to cross the virtualboundary and leave the activity space. Some visual cues may for examplebe displayed to prompt him to stay in the activity space to enjoy thebest user's experience. In the case the user moves across the boundarywithout respecting the warnings, the mixed-reality application (runtimeapplication) could for example be paused until the user moves back tothe activity space.

The previously introduced embodiments regarding generating the augmentedreality representation are now further detailed.

In one embodiment, the scale of the virtual objects is adapted as afunction of the mixed space of the real environment. More particularly,a default virtual scene comprising the virtual objects to be displayedis compared with the mixed space. The comparison is made on the basis ofa size difference such as, for example, a difference between a virtualground area size of the default virtual scene and a real ground areasize of the mixed space of the real environment. The real ground areasize may be obtained as an output of real scene analysis or layoutdetection, as already described in relation with FIG. 4. The area sizeof the largest horizontal plane detected in the mixed space (other thana ceiling, e.g. the surface of the real ground or of a table) is forexample considered as a reference scale for the mixed space. Dependingon the augmented reality application, the virtual ground area may beexplicitly configured in the application, or not. If the virtual groundarea is configured in the augmented reality application, its size isdirectly available. If not, this size is estimated by projecting thevirtual objects belonging to the default virtual scene and theirtrajectories on a horizontal plane, and by computing the size of theconvex hull encompassing the projected elements. An example of such acase is illustrated in relation with FIG. 7. The virtual scene comprisestwo statics virtual objects V1 and V3, and a moving virtual object V2.The area size of the convex hull VG encompassing virtual objects V1 andV3, initial and final positions of virtual object V2, and the trajectoryTV2 of V2 between these initial and final positions, is used as areference scale for the default virtual scene.

Depending on the ratio between the reference scale for the defaultvirtual scene and the reference scale for the mixed space, the defaultvirtual scene is scaled down, scaled up, or not scaled. According to aparticular feature, the default virtual scene is scaled (if necessary)so that the ratio between the scaled virtual scene and the referencescale for the mixed space is comprised between one third and two-thirds(meaning that the area size of the scaled virtual ground is larger thanone third of the area size of the considered real ground of the mixedspace, and smaller than two-thirds of said area size of the consideredreal ground of the mixed space). Such a ratio is indeed considered as anappropriate ratio which allows, in most cases, a realistic insertion ofthe virtual scene within the real scene.

Of course, managing the virtual content is not limited to scaling thevirtual scene, but also includes for example obtaining the optimallocation for the virtual objects according to the layout of the realscene and identified objects in the mixed space, and adjusting thevirtual events accordingly (i.e. managing interaction between virtualobjects and specific real objects of the mixed space). It should also benoted that although the configuration (i.e. scale, location,interaction) of virtual content only depends on the real objects presentin mixed space, the rendering of virtual content should however, undercertain circumstances, be determined as a function of a representationof at least one real content in the activity part. For example,referring back to the augmented reality representation AR of FIG. 3,virtual object V3 is partially obscured by real object R2, to keep theaugmented reality representation realistic. In other words, virtualobjects are to be rendered in the mixed part according to theirocclusion with real and other virtual objects present in the mixed part,but also according to their occlusion with real objects present in theactivity part.

As explained throughout the document, the disclosure relies on thevirtual division of the real environment into two distinct spaces: theactivity space assimilated as a safety zone within which a user can movearound safely while viewing an augmented representation of the realenvironment, and the mixed space more specifically defined as the usablespace for the insertion of the virtual content in the generatedaugmented reality representation. The user is thus encouraged to stay inthe activity space while viewing the augmented reality representation,to enjoy an optimal augmented reality experience. Thus, according to anembodiment, the user moving around within the activity space is alertedwhen a distance between said user and the virtual boundary is lower thana predetermined threshold. The alert may take the form of audio alarm,or of virtual cues warning displayed on the augmented realityrepresentation. According to a particular feature, the augmented realityapplication is paused if the user ignores the warnings and breaks intothe mixed space (and may be restored when the user moves back to theactivity space).

Device

According to another aspect, the present disclosure also pertains to adevice for generating an augmented representation of a real environment,called an augmented reality representation, wherein said devicecomprises at least one processor adapted and configured for:

-   -   obtaining, as a function of said real environment, a virtual        boundary dividing said real environment into two spaces,        respectively called an activity space and a mixed space;    -   generating said augmented reality representation, comprising:        -   obtaining a first part of said augmented reality            representation, called an activity part, corresponding to a            representation of at least one part of said activity space;        -   obtaining a second part of said augmented reality            representation, called a mixed part, corresponding to an            augmented representation of at least one part of said mixed            space, in which at least one virtual content is combined            with said representation of at least one part of said mixed            space.

Such a device can be especially adapted to implement the method forgenerating an augmented representation of a real environment describedhere above. It could of course comprise the different characteristicspertaining to the method according to an embodiment of the disclosure,which can be combined or taken separately. Thus, the characteristics andadvantages of the device are the same as those of the method.

FIG. 8 is a schematic block diagram illustrating an example of a device800 for generating an augmented representation of a real environmentaccording to an embodiment of the present disclosure. In an embodimentof the proposed technique, such a device may be embedded in an apparatussuch as a head-mounted display, a mobile phone, or a tablet for example.In another embodiment, it may be an external device connected to or incommunication with a rendering apparatus able to display an augmentedreality representation to a user.

The device 800 includes a processor 801, a storage unit 802, an inputdevice 803, an output device 804, and an interface unit 805 which areconnected by a bus 806. Of course, constituent elements of the computerdevice 800 may be connected by a connection other than a bus connectionusing the bus 806.

The processor 801 controls operations of the device 800. The storageunit 802 stores at least one program to be executed by the processor801, and various data, including for example data related to the userand data related to the real environment, parameters used bycomputations performed by the processor 801, intermediate data ofcomputations performed by the processor 801, and so on. The processor801 is formed by any known and suitable hardware, or software, or acombination of hardware and software. For example, the processor 801 isformed by dedicated hardware such as a processing circuit, or by aprogrammable processing unit such as a CPU (Central Processing Unit)that executes a program stored in a memory thereof.

The storage unit 802 is formed by any suitable storage or means capableof storing the program, data, or the like in a computer-readable manner.Examples of the storage unit 802 include non-transitorycomputer-readable storage media such as semiconductor memory devices,and magnetic, optical, or magneto-optical recording media loaded into aread and write unit. The program causes the processor 801 to perform amethod for generating an augmented representation of a real environmentaccording to an embodiment of the present disclosure as describedpreviously. More particularly, the program causes the processor 801 toobtain, as a function of the real environment, a virtual boundarydividing said real environment into two spaces, respectively called anactivity space and a mixed space and to take into account this spacedivision when generating an augmented representation of the realenvironment.

The input device 803 is formed for example by one or several sensorsallowing determining various data related to the user and various datarelated to the real environment.

The output device 804 is formed for example by a head-mounted display, atablet display, or a mobile phone display to display the augmentedreality representation generated by applying the method previouslydescribed.

The interface unit 805 provides an interface between the device 800 andan external apparatus. The interface unit 805 may be communicable withthe external apparatus via cable or wireless communication. For example,in an embodiment where the device 800 is not embedded in a head-mounteddisplay, a tablet or a mobile phone, an external apparatus may berespectively such a head-mounted display, tablet or mobile phone.

Although only one processor 801 is shown on FIG. 8, it must beunderstood that such a processor may comprise different modules andunits embodying the functions carried out by device 800 according toembodiments of the present disclosure, for generating an augmentedrepresentation of a real environment, such as:

-   -   a module for obtaining, as a function of said real environment,        a virtual boundary dividing said real environment into two        spaces, respectively called an activity space and a mixed space;    -   a module for generating said augmented reality representation,        comprising:    -   a module for obtaining a first part of said augmented reality        representation, called an activity part, corresponding to a        representation of at least one part of said activity space;    -   a module for obtaining a second part of said augmented reality        representation, called a mixed part, corresponding to an        augmented representation of at least one part of said mixed        space, in which at least one virtual content is combined with        said representation of at least one part of said mixed space.

The processor 801 may of course also comprise other modules allowingimplementing the functions corresponding to the various otherembodiments previously described in relation with the method forgenerating an augmented representation of a real environment.

These modules and units may also be embodied in several processors 801communicating and co-operating with each other.

Computer Program Product and Non-Transitory Computer-Readable Medium

Another aspect of the present disclosure pertains to at least onecomputer program product downloadable from a communication networkand/or recorded on a medium readable by a computer and/or executable bya processor, comprising program code instructions for implementing themethod as described above.

In addition, the present disclosure also concerns a non-transitorycomputer-readable medium comprising a computer program product recordedthereon and capable of being run by a processor, including program codeinstructions for implementing the method for generating an augmentedrepresentation of a real environment as described above.

The computer readable storage medium as used herein is considered anon-transitory storage medium given the inherent capability to store theinformation therein as well as the inherent capability to provideretrieval of the information therefrom. A computer readable storagemedium can be, for example, but is not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Itis to be appreciated that the following, while providing more specificexamples of computer readable storage mediums to which the presentprinciples can be applied, is merely an illustrative and not exhaustivelisting as is readily appreciated by one of ordinary skill in the art: aportable computer diskette; a hard disk; a read-only memory (ROM); anerasable programmable read-only memory (EPROM or Flash memory); aportable compact disc read-only memory (CD-ROM); an optical storagedevice; a magnetic storage device; or any suitable combination of theforegoing.

1. A method for generating an augmented representation of a realenvironment comprising at least one real object, called an augmentedreality representation, the method comprising: obtaining, as a functionof data associated with said at least one real object of said realenvironment, a virtual boundary dividing said real environment into twospaces, respectively called an activity space and a mixed space;generating said augmented reality representation, comprising: obtaininga first part of said augmented reality representation, called anactivity part, corresponding to a representation of at least one part ofsaid activity space, said activity part being free of virtual content;obtaining a second part of said augmented reality representation, calleda mixed part, corresponding to an augmented representation of at leastone part of said mixed space, in which at least one virtual content iscombined with said representation of at least one part of said mixedspace.
 2. The method according to claim 1, wherein said virtual boundaryis obtained as a function of a shape of said at least one real object.3. The method according to claim 1, wherein said at least one realobject is a moving object, and wherein said virtual boundary is obtaineddynamically, as a function of said at least one moving real object. 4.The method according to claim 1, wherein obtaining said virtual boundarycomprises obtaining said virtual boundary as a function of a category ofsaid real environment.
 5. The method according to claim 1, whereinobtaining said virtual boundary comprises obtaining at least part ofsaid virtual boundary as a function of a difference between a scale ofsaid real environment and a scale of a default virtual scene comprisingsaid at least one virtual content.
 6. The method according to claim 1,wherein said data associated with said at least one object comprise aposition of a user of said augmented reality representation within saidreal environment.
 7. The method according to claim 1, wherein said dataassociated with said at least one object comprise a current heightand/or a current posture of a user of said augmented realityrepresentation.
 8. The method according to claim 1, wherein obtainingsaid mixed part of the augmented reality representation comprisesscaling said at least one virtual content as a function of at least onedifference between a size of a default virtual scene comprising said atleast one virtual content and a size of said real environment in saidmixed space.
 9. The method according to claim 8, wherein said at leastone virtual content is scaled as a function of a difference between asize of a virtual ground area of said default virtual scene and a sizeof a real ground area of said real environment in said mixed space. 10.The method according to claim 1, wherein at least one virtual content insaid mixed part is obtained by taking account of a representation ofsaid at least one real object in said activity part.
 11. The methodaccording to claim 1, wherein it comprises alerting a user of saidaugmented reality representation when a distance between said user andsaid boundary is lower than a predetermined threshold.
 12. Anon-transitory computer-readable medium comprising a computer programproduct recorded thereon and capable of being run by a processor,including program code instructions for implementing a method accordingto claim
 1. 13. A device for generating an augmented representation of areal environment comprising at least one real object, called anaugmented reality representation, the device comprising at least oneprocessor adapted and configured for: obtaining, as a function of dataassociated with said at least one real object of said real environment,a virtual boundary dividing said real environment into two spaces,respectively called an activity space and a mixed space; generating saidaugmented reality representation, comprising: obtaining a first part ofsaid augmented reality representation, called an activity part,corresponding to a representation of at least one part of said activityspace, said activity part being free of virtual content; obtaining asecond part of said augmented reality representation, called a mixedpart, corresponding to an augmented representation of at least one partof said mixed space, in which at least one virtual content is combinedwith said representation of at least one part of said mixed space. 14.The device according to claim 13, wherein said virtual boundary isobtained as a function of a shape of said at least one real object. 15.The device according to claim 13, wherein said at least one real objectis a moving object, and wherein said virtual boundary is obtaineddynamically, as a function of said at least one moving real object. 16.The device according to claim 13, wherein obtaining said virtualboundary comprises obtaining said virtual boundary as a function of acategory of said real environment.
 17. The device according to claim 13,wherein obtaining said virtual boundary comprises obtaining at leastpart of said virtual boundary as a function of a difference between ascale of said real environment and a scale of a default virtual scenecomprising said at least one virtual content.
 18. The device accordingto claim 13, wherein said data associated with said at least one objectcomprise a position of a user of said augmented reality representationwithin said real environment.
 19. The device according to claim 13,wherein said data associated with said at least one object comprise acurrent height and/or a current posture of a user of said augmentedreality representation.
 20. The device according to claim 13, whereinobtaining said mixed part of the augmented reality representationcomprises scaling said at least one virtual content as a function of atleast one difference between a size of a default virtual scenecomprising said at least one virtual content and a size of said realenvironment in said mixed space.
 21. The device according to claim 20,wherein said at least one virtual content is scaled as a function of adifference between a size of a virtual ground area of said defaultvirtual scene and a size of a real ground area of said real environmentin said mixed space.
 22. The device according to claim 13, wherein atleast one virtual content in said mixed part is obtained by takingaccount of a representation of said at least one real object in saidactivity part.
 23. The device according to claim 13, comprising anoutput device for alerting a user of said augmented realityrepresentation when a distance between said user and said boundary islower than a predetermined threshold.